Augmentation of captured 3d scenes with contextual information

ABSTRACT

Augmentation of captured 3D scenes with contextual information is disclosed. A 3D capture device is used to capture a plurality of 3D images at a first resolution. A component on a mobile computing device is used to capture at least one piece of contextual information that includes a capture location data and a pose data. The mobile computing device receives, the plurality of 3D images from the 3D capture device, and renders the plurality of 3D images into a 3D model. In addition, the at least one piece of contextual information is embedded into a correct location in the 3D model. A user interactive version of the 3D model including the embedded at least one piece of contextual information is then displayed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/980,913, filed May 16, 2018, entitled “Augmentation Of Captured 3DScenes With Contextual Information” by Rafael Spring et al., assigned tothe assignee of the present application, having Attorney Docket No.DP-005, which is herein incorporated by reference in its entirety.

U.S. patent application Ser. No. 15/980,913 claims benefit of U.S.Provisional Patent Application Ser. No. 62/507,494, filed May 17, 2017,entitled “User-Friendly Augmentation Of Captured 3D Scenes WithContextual Information” by Rafael Spring, assigned to the assignee ofthe present application, having Attorney Docket No. JPEX-DP-005.PRO,which is herein incorporated by reference in its entirety.

FIELD OF THE INVENTION

Embodiments of the invention generally relate to three-dimensional datacapture of scenes.

BACKGROUND

In many industries, 2D pictures are taken as a form of documentation ofa scene, object, or asset. Often 100s of pictures will be taken, forexample at a crime scene, accident scene, or a construction site, or anindustrial facility or ship that needs repairs or renovations orvalidation that it has been constructed according to the design models.However, it is very difficult for users to understand where the pictureswere taken and from what viewpoint. This can severely limit theusefulness of 2D pictures in these applications.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and form a part ofthis specification, illustrate various embodiments and, together withthe Description of Embodiments, serve to explain principles discussedbelow. The drawings referred to in this brief description should not beunderstood as being drawn to scale unless specifically noted.

FIG. 1 is a block diagram of a handheld portable optical scanningsystem, in accordance with an embodiment.

FIG. 2 is a flowchart for capturing additional contextual informationwhile capturing 3D information, in accordance with an embodiment.

FIG. 3 is a flowchart for the display and modification of captured 3Dscenes augmented with contextual information, in accordance with anembodiment.

FIG. 4A is a screen shot of an initial image view of the scene capturingprocess presented on a display screen in accordance with an embodiment.

FIG. 4B is a screen shot of 3D capture view presented on the displayscreen in accordance with an embodiment.

FIG. 4C is a screen shot of a contextual information capture ready viewpresented on the display screen in accordance with an embodiment.

FIG. 4D is a screen shot of a successful contextual information captureview presented on the display screen in accordance with an embodiment.

FIG. 4E is a screen shot of a 3D scene preview prior to optimizationpresented on the display screen in accordance with an embodiment.

FIG. 4F is a screen shot of an optimized 3D scene including a view toolbar presented on the display screen in accordance with an embodiment.

FIG. 4G is a screen shot of an optimized 3D scene rendered withcontextual information identifiers presented on the display screen inaccordance with an embodiment.

FIG. 4H is a screen shot of an optimized 3D scene rendered withcontextual information identifiers and their associated image framespresented on the display screen in accordance with an embodiment.

FIG. 4I is a screen shot of an optimized 3D scene with an addedreference coordinate system presented on the display screen inaccordance with an embodiment.

FIG. 4J is a screen shot of an optimized 3D scene showing an in-scenemeasurement capability presented on the display screen in accordancewith an embodiment.

FIG. 4K is a screen shot of an optimized 3D scene with added annotationpresented on the display screen in accordance with an embodiment.

FIG. 5 is a block diagram of an example computer system with which orupon which various embodiments of the present invention may beimplemented.

DESCRIPTION OF EMBODIMENTS

Reference will now be made in detail to embodiments of the subjectmatter, examples of which are illustrated in the accompanying drawings.While the subject matter discussed herein will be described inconjunction with various embodiments, it will be understood that theyare not intended to limit the subject matter to these embodiments. Onthe contrary, the presented embodiments are intended to coveralternatives, modifications and equivalents, which may be includedwithin the spirit and scope of the various embodiments as defined by theappended claims. Furthermore, in the Description of Embodiments,numerous specific details are set forth in order to provide a thoroughunderstanding of embodiments of the present subject matter. However,embodiments may be practiced without these specific details. In otherinstances, well known methods, procedures, components, and circuits havenot been described in detail as not to unnecessarily obscure aspects ofthe described embodiments.

Notation and Nomenclature

Unless specifically stated otherwise as apparent from the followingdiscussions, it is appreciated that throughout the present Descriptionof Embodiments, discussions utilizing terms such as “selecting”,“outputting”, “inputting”, “providing”, “receiving”, “utilizing”,“obtaining”, “updating”, “accessing”, “changing”, “correlating”,“prescreening”, “developing”, “presenting”, “deploying” or the like,often refer to the actions and processes of an electronic computingdevice/system, such as a desktop computer, notebook computer, tablet,mobile phone, and electronic personal display, among others. Theelectronic computing device/system manipulates and transforms datarepresented as physical (electronic) quantities within the circuits,electronic registers, memories, logic, and/or components and the like ofthe electronic computing device/system into other data similarlyrepresented as physical quantities within the electronic computingdevice/system or other electronic computing devices/systems.

Overview

Importantly, the embodiments of the present invention, as will bedescribed below, provide a process for capturing additional contextualinformation for objects or features of interest directly while alsocapturing 3D data using an interface displayed on a hand-held mobiledevice. In so doing, the user sees a real-time visualization of the datathey are capturing, as a visualization of a 3D model building up as thedata capturing progresses. In addition, in one embodiment, theadditional contextual information is embedded into the 3D model anddisplayed within the 3D model on the mobile device display as aninteractive display. This approach differs significantly from theconventional processes used for 3D scene recording, rendering,processing, and post-processing user interaction.

The capturing and modeling of an environment in a 3D scene is importantfor modeling power utility rooms, vessel compartments, etc. For example,a location such as a mechanical room with pumps, generators, electriccontrol boxes, etc. where there is no present diagram, or floor plan orCAD model showing what is actually in the room, any modifications madeto things in the room, updates, etc.

3D scene capture is also valuable to determine an as built conditionversus the as designed version of a building, room, etc. It can also bevaluable in providing an actual layout of buildings, rooms, etc. Forexample, as one or more rooms are updated the information changes. If anew design is being considered, a 3D model of the actual space would beimportant for determining material amounts, costs, what would fit andwould not fit, etc.

The addition of contextual information into the 3D model is additionallyvaluable in areas such as a crime scene, where, when photos are taken,it can be difficult if not impossible to manage the organization of thephotos (e.g., the where, the size, the direction, etc.) to make sure thelayout of the photos is correct. By augmenting the captured 3D sceneswith contextual information, the photos taken at the scene are tagged asbeing in a specific location and then are stitched into the scene in theappropriate location with the appropriate pose data. Moreover, theconglomeration can resolving ambiguities about which witnesses can seewhat from where and when, which can require robust 3D geometric contextnot obtainable with 2D images only. In one embodiment, pose dataincludes a point in 3D space (e.g. XYZ values) and a directionalcomponent in 3D space (e.g. rotation matrix or quaternion) for thecontextual information at a time the at least one piece of contextualinformation was captured (e.g., direction, angle, pitch, yaw, etc.). Inanother embodiment, pose data includes a reference frame in the 3D sceneand a pose (3D position and rotation) relative to the reference frame'spose.

In general, each pixel in the 3D image includes a depth value such thatreal scale can be defined in the 3D scene that is modeled. The depthvalue could utilize metric measurements, English measurements, or thelike. As such, embodiments described herein are also valuable forobtaining measurement information in a 3D model of congested spaces,spaces that include items that are too complex to measure by hand,spaces that do not have enough room to get all the measurements, spaceswith numerous layers of pipes, cords, wires, etc. that are not easilyseparated, and the like.

In one embodiment, the contextual information can be 2D images that arehigher resolution than the 3D images being captured, audio information,a higher resolution video snippet, etc. In one embodiment, thecontextual information is added to the 3D model and represented as anicon or link within the 3D scene. Moreover, the icon or link can beselected by a user interacting with the 3D model to provide thecontextual information to the user. For example, the contextualinformation captured could be a tag, an easier interface, a clearerview, an area of interest, an audio capture, a higher resolution videocapture (such as a view out of a window or door, people in the crowd, astatement, etc.), or the like as described in further detail herein.

In conventional approaches, the 3D model/point cloud is obtained as partof a first step and then a second post processing step is performed on adifferent more powerful computer which loads up the 3D model (or partsof it) into point cloud processing software which will performfully-automatic conversion of the point cloud into geometric primitivesor allows a user to hand-select certain sections of the point cloud andlet the software perform primitive detection/conversion on thoseselected parts (semi-automatic). Both fully-automatic and semi-automaticapproaches are time-consuming and challenging to the normal user. Suchconventional approaches are error prone, tedious, time-consuming, andoftentimes include significant delay between when the data is capturedand when it is ready to be presented.

Instead, the present embodiments, as will be described and explainedbelow in detail, provide a previously unknown procedure for generating a3D model, while also allowing the embedding of any captured contextualinformation (e.g., a 2D higher resolution image, a higher resolutionvideo snippet, an audio file, or the like) into the 3D model, at thetime of capture or afterwards. By embedding contextual information in a3D model and giving the viewer of the 3D model the ability to seamlessly“fly” between different embedded contextual information in the scene,one embodiment solves the problem of regular 2D pictures lacking 3Dvisual context. Instead, the user is able to get a spatial understandingof where the contextual information was obtained and how they relate toeach other in 3D, which extends well beyond what was previously capable.

As will be described in detail, the various embodiments of the presentinvention do not merely implement conventional processes for capturingand building a 3D scene. Instead, the various embodiments of the presentinvention, in part, provide a previously unknown procedure forincorporating additional contextual information at a higher resolutioninto the 3D scene. Hence, embodiments of the present invention provide anovel process for 3D modeling which is necessarily rooted in computertechnology to overcome a problem specifically arising in the realm ofreal-time 3D capturing and modeling.

Moreover, the embodiments do not recite a mathematical algorithm; theydo not recite a fundamental economic or longstanding commercialpractice; and they do not merely recite the performance of some practiceknown from the pre-Internet world along with the requirement to performit on the Internet or on a computer. Instead, the embodiments arenecessarily rooted in 3D modeling technology in order to overcomeproblems specifically arising in the realm thereof.

Operation

With reference now to FIG. 1, a block diagram of a handheld portableoptical scanning system 100 is shown in accordance with an embodiment.System 100 includes a mobile device 110. Although FIG. 1 shows ahandheld portable optical scanning system of structured-IR-light-type,it should be appreciated that the 3D capture device could be any of aplurality of possible handheld portable scanning systems. The use of thesystem shown in FIG. 1 is provided as one example option.

In general, mobile device 110 may be a mobile phone, a smart phone, atablet, a laptop, a smart watch, a piece of smart jewelry, smartglasses, and other user portable and/or wearable computing devices. Inone embodiment, mobile device 110 has wireless connectivity. That is,mobile device 110 is capable of broadcasting and receiving via at leastone network, such as, but not limited to, WiFi, Cellular, Bluetooth,NFC, and the like. In one embodiment, mobile device 110 will have apositioning determining system such as global positioning system (GPS)or the like. In another embodiment, mobile device 110 is able todetermine a location within a given radius, such as the broadcast rangeof a beacon, WiFi hotspot, overlapped area covered by a plurality ofmobile telephone signal providers, or some combination thereof. Further,the location determined by mobile device 110 could be useful even atdiffering levels of accuracy. For example, a GPS enabled mobile devicecould provide location information that is accurate to within a fewinches while a cellular service, beacon or WiFi location capabilities ofmobile device 110 could provide a location radius or location area. Forexample, mobile device 110 being located within range of a beacon,within the overlapping area of a number of cellular service towers, etc.

With reference still to FIG. 1, in one embodiment the user desires tocapture a scene of interest 101. The 3D video camera 102 is used forcapturing multiple images of a scene along with a depth map for eachscene. Mobile device 110 is used for implementing a 3D rendering systemand operating the complete system. Mobile device 110 is configured toenable a user to operate the image capture device, to obtain a datastream of images with depth map information for the image, which mayinclude depth information for pre-determined spots in the image, and toperform any of a number of image manipulations based on additionalsoftware available to the tablet/smartphone computer. Mobile device 110has programs for its internal operations and applications programs formanaging the image capture and various processing tasks. Software forimproved image processing applications 140 includes programs that cancapture and deliver image and depth information, combine multiple imagesinto a single 3D model for viewing, analyzing, and managing. Softwarefor image and model manipulation and management in real time, includesimage management 160, remote operations 180 and imager calibration 190.Additional software is available for performing many other functions onthe captured and combined images, for a variety of purposes.

In one embodiment, 3D video camera 102 is an image capture and depth mapdeterminer. For example, in one embodiment, 3D video camera 102 includesa red-green-blue-depth (RGB-D) camera as the principal sensor, operatedby and in conjunction with Mobile device 110. The 3D video camera 102 isconfigured to operate with a structured-light depth map determinationsystem. Alternatively, the 3D video camera 102 may be configured tooperate with a time-of-flight depth map generator. Alternatively, the 3Dvideo camera 102 may be configured to operate with a stereo orassisted-stereo depth map generator. Alternatively, the 3D video camera102 may be configured to operate with a MEMS-mirror- orsolid-state-LiDAR depth map generator. Sensors provide depth map data inreal time using inexpensive components.

In one embodiment, a specialized infrared light beam system 103broadcasts a dot pattern toward a scene of interest, delivering a seriesof tiny dots that illuminate the objects in the scene. The number andspacing of the dots defines the resolution obtainable. An imager 104,similar to what is found in digital cameras, captures the scene and theilluminating dots in a special sensor, called the PS1080. Imager 104includes an image capture lens and a CMOS sensor. The sensor derives asynchronized depth image, a color image, and optionally an audio datastream.

3D video camera 102 may also include a custom processor 105, which mayinclude a control portion 106 and storage 107 for color and depthinformation.

Similarly, fully integrated sensors 108 for performing time-of-flight(TOF) distance measurements without any moving parts are available fromthe PMD Technologies Co. in Siegen, Germany. More information aboutthese systems may be found at www.pmdtec.com. The sensor generates amodulated optical signal, and measures time of flight directly.

For example, the PMD PhotonlCs 19 k-S3 chipset obtains distancemeasurements to each pixel instantly, thus providing both a 3Drepresentation of each pixel in view in the scene, as well as grey scaleillumination data, simultaneously. Data from the chipset may be read outat rates of 15 MPixels/second. PMDTech also offers a complete camerasystem called the CamBoard, which is the first USB powered single board3D TOF camera. Other companies with similar products include SoftKineticand MESA Imaging. The capture rate for these sensors permits image framecapture at rates up to 60 frames/second (fps). These sensors do notprovide the same level of resolution that more complicated and moreexpensive scanners can provide.

In one embodiment, Mobile device 110 serves as the user interface forcontrolling the image sensor and depth capture sensor subsystem of 3Dvideo camera 102. Mobile device 110 can be operated by holding it in oneor two hands, without any additional support; to be able to see theresultant two-dimensional (2D) image as captured by the 3D video camera102, on a display 116; and to be able to input control information andcommands via either a touch screen (also at 116) or an optional keyboardat 117. An audio output 118 is also optional. The processor 111available in tablet computers has suitably fast clock operations, tofacilitate real time operation of the image/depth capture system andprocess the image and depth data, to provide a visible image innear-real to real time. Additional features and functions common in mostif not all of such handheld computers available today and connected onbus 112 may include a second internal camera 113, a communicationssystem 114 further comprising at least one of a cellular telephony link,a cellular data link, and a Wi-Fi link.

Software such as operating system 130 contains applications foroperating these accessory functions, along with data management andstorage in ROM 119, RAM 120, and data storage 121, which may comprise anexternal memory device like a USB memory stick, or any other suitablenon-volatile storage medium. Besides the operating system, software mayinclude image processing applications 140, image and image management160, and a suite of software for imager calibration 190. In oneembodiment, each of these may include a variety of separate programs. Inan embodiment of the present disclosure, audio capture via the customprocessor 105 and audio playback via software in the operating system130 enable capture and playback of sounds during image capture as well.This feature facilitates verbal note-taking while performing the imagedata capture if so desired. In one embodiment, a local positioningsystem 115 or aiming system may also be used.

In one embodiment, image processing applications 140 are provided forusing a stream of RGB-D video frames to form the combined 3D data set.These include Image Capture and Pre-processing programs as one of thegroup of image processing applications 140, the Computer Vision andScanning suite. For capturing and rendering, the suite includes areal-time RGB-D image visualization program. The software will operateon mobile device 110.

In an embodiment, new stitching or combining software is used toautomatically merge two or more images together to form a composite 3Dmodel using aspects such as the use of keyframes as discussed in moredetail herein. With this software tool, a model may be created from oneor more images taken from different viewpoints in the scene. Thisprocess is done in real-time, on-the-fly, so that the user can, at anytime, view the formed 3D model, even during capturing, from a variety ofviewpoints. This includes the current viewpoint of the attached camera,resulting in an augmented-reality-style visualization. The instant 3Dmodel formation enables a user to see where additional contextualinformation may be useful, and enables the user to point the camera tothe desired region to obtain the contextual information.

Elements of the software include suggestions for user-initiated actionsto complete a portion of a scanned image, including directions to aimthe 3D video camera 102. Because of the power of the combiningalgorithms used, including the capability of 3D reconstruction, imagesobtained from other instruments with differing levels of detail may alsobe inputted into the mobile device 110. The advantage of fast combiningor modeling means that field adjustments and retakes can be done in nearreal-time with these other instruments as well as with the instantsystem. For example, image capture devices which also produce 3D depthmaps along with greyscale or color images may be inputted to thissystem.

In one embodiment, the software provides an integrity metric to indicatewhen there is not enough data to perform a decent combining or 3-Dmodeling operation or to obtain registration of particular image with apreviously declared registration point in the image. The declaredregistration point may be obtained from an arbitrary model, either froma computer-aided design (CAD) model or a 3D point cloud model. The userinterface is changed in a way that the user sees or is notified wherethere is not enough data captured in the scene as the scene is beingcombined or modeled.

With reference now to FIG. 2, a flowchart 200 for capturing additionalcontextual information while capturing 3D information is shown inaccordance with an embodiment.

At 205 of FIG. 2, the user will start the 3D capture. An example of thestart of the capture is shown in FIG. 4A which provides a screen shot400 of an initial image view presented on a display 116 in accordancewith an embodiment. As shown in screen shot 400, the view presented ondisplay 116 includes crosshairs 402, main menu 403, and scan start/stop401.

Crosshairs 402 is a displayed graphic that that gives the user theability to tag whatever is under the crosshair at the time of capture atthe push of a button, or other user command (e.g. spoken command) ordetermined automatically by the software based on a variety of otherfactors (e.g. motion of the capture device, time, etc.).

In one embodiment, to perform the tagging, the 3D capture device posetracking technology (e.g. as provided by the DotProduct DPI-8 or theLenovo Phab 2 Pro products) provides the application developer withcontinuous camera pose estimates (spatial position and orientation),typically at a rate of equal to or higher than one pose per cameraframe.

The camera pose at the time a tagging event occurs can be determinedusing 3D pose tracking technology. Information from a software interfaceor as loaded from a calibration file is used to obtain the capturecamera intrinsic information (focal lengths, optical center, lensdistortion parameters).

The camera pose at the time of the tagging event, the camera intrinsicparameters and the position of the crosshair uniquely determine aposition and direction vector in space at the time of capture by: p=t,d=R*uncam(c) where p is the position vector, t is the translation partof the camera pose matrix Rt, d is the direction vector, c is thecrosshair position in camera pixel coordinates, R is the rotation partof the camera pose matrix and uncam( ) is a function that takes camerapixel coordinates to normalized camera coordinates. If necessary, theobtained position and direction vectors can then be intersected with thegeometry of the captured scene to obtain the tag position in the globalframe.

For example, a set of 3D points in the current captured scene using thecrosshair 402 position and either the currently captured depth map (incase a depth sensor is the capturing unit) or the currently capturedpoint cloud (in case a LiDAR based sensor is the capturing unit), thepoint set obtained by querying the depth map/point cloud for points inthe neighborhood of the crosshair position, or the 3D model currentlyunder capture, the point set obtained by querying the 3D model forpoints in the neighborhood of the tag position. Then fit a geometricprimitive or parametric shape to the point set using shape fittingtechniques. Then, by knowing the camera pose as well as the primitive orshape pose in the camera frame the primitive or shape pose in the globalframe can be computed by multiplying the local primitive or shapecoordinates with the pose matrix.

In one embodiment, the 3D video camera 102 will start the capture whenscan start/stop 401 is selected. Once scan start/stop 401 is selectedthe 3D video camera 102 works by grabbing frames of points and withineach frame the depth associated with each point. In one embodiment, theinitial starting point of 3D video camera 102 could be assigned thevector (0,0,0). In another embodiment, the initial starting point of 3Dvideo camera 102 could be assigned coordinates determined by a GPS.

In one embodiment, 3D video camera 102 keeps track of its location basedon the spatial relationship of the current camera frame with alreadycaptured imagery. For example, as 3D video camera 102 moves from theinitial starting point (e.g., moved rotationally, moved around the room,a combination of rotation and movement, pitch, yaw, roll, a combinationof any/or all of the motions, etc.), it keeps track of its position bycomparing elements in the current camera image to previous cameraimages. In one embodiment, the comparison is performed by akeyframe-based method that uses a stitching operation that aligns two ormore RGB-D images as described herein. In another embodiment thecomparison is performed by an off-the-shelf camera pose tracking system,such as found in mobile devices using Apple™ Inc's ARKit™ or Google™Inc's ARCore™ or Tango™ technologies. In general, the pose trackingallows the 3D model to be developed and the location/look direction,etc. of the 3D video camera 102 to be determined without requiring 3Dvideo camera 102 or mobile device 110 to maintain or obtain anyadditional location information.

In one embodiment, if the 3D video camera 102 is turned off during thecapturing process, is moved to a different position too quickly or movedto scenes unsuitable for pose tracking (such as scenes lacking a minimumof structure and texture) or if it's otherwise not possible for thetracking to relate the current frame content to the previously capturedframes, the system will report the loss of tracking. In one embodiment,to realign/reorient the “lost” 3D video camera 102, it can be aimed at apreviously captured section and then continue to be moved about the areato be captured and modeled. In so doing, the newly captured imagery willbe compared with the previously captured imagery and the 3D video camera102 will be re-localized.

In one embodiment, if the scene being captured is bland (e.g., lacking aminimum of structure and texture), it may be necessary to return the 3Dvideo camera 102 to an obvious geometry, color, or other identifierwithin the scene that will provide enough information to allow there-localization to occur. In one embodiment, during the capture processof the 3D scene, the redundant information captured when performing there-localization would be ignored.

At 210 of FIG. 2, one embodiment determines if there is any interestingcontextual information that has not been tagged. For example, during thecapturing of the imagery by the 3D video camera 102, there may be asection, portion, location, item, or the like that a user would like tocapture in addition to the 3D imagery being captured.

At 220 of FIG. 2, when it is determined that there is interestingcontextual information that has not been tagged, the contextualinformation is captured. For example, referring to FIG. 4B, a screenshot 405 of a 3D capture view presented on the display 116 is shown inaccordance with an embodiment. Screen shot 405 includes crosshairs 402,contextual information capture 407, scan start/stop 401, and large ring408. In general, large ring 408 is used to indicate to the user thatthere is too much motion for the contextual information capturing device(e.g., a 2D camera, a 2D video camera, a sound recorder, or the likewhich may be part of/or a module added to mobile device 110) to obtain a2D image (or 2D video, sound recording, etc.). In one embodiment thecontextual capturing device may be of higher resolution than what isalready embedded in the 3D capture device.

Referring now to FIG. 4C, a screen shot 410 of a contextual informationcapture ready view is presented on the display 116 in accordance with anembodiment. Screen shot 410 includes crosshairs 402, contextualinformation capture 407, scan start/stop 401, and small ring 412. Ingeneral, small ring 412 is used to indicate to the user that the motionhas been reduced to within the bounds of contextual informationcapturing device's ability to capture a higher resolution image (orvideo). Once large ring 408 disappears and small ring 412 appears theuser can select contextual information capture 407 and capture thecontextual information.

With reference now to FIG. 4D, a screen shot 415 of a successfulcontextual information capture is presented on the display 116 inaccordance with an embodiment. Screen shot 415 includes crosshairs 402,contextual information capture 407, scan start/stop 401, small ring 412,and information ribbon 416. In general, information ribbon 416 indicatesthe contextual information has been successfully captured.

In other words, the 3D video camera 102 is held relatively still,contextual information capture 407 is selected and the contextualinformation is captured. Although a 2D higher resolution image isdiscussed in a number of examples herein as the contextual information,it should be appreciated that the contextual information that iscaptured could be any or all of a 2D image, a sound recording, a videorecording (e.g., a video snippet), and the like, whereby the resolutionof an image camera or video camera may be equal to or higher than theresolution of the image- or video-camera already embedded in theimage/depth capture module 102. However, depending upon the size ornecessary clarity of the contextual information to be captured, theequal or higher resolution could be selected from resolutions such as,but not limited to, HD, 1080p, 2K, 4K, 8K, 64K, 2Mpix, 5Mpix, 10Mpix,51Mpix, etc.

In one embodiment, instead of (or in addition to) a user having toactively perform the collection of the contextual information (e.g., tapcontextual information capture 407) a number of different collectionmethods could be used. For example, during the 3D video camera 102capturing process, if the 3D video camera 102 is aimed at the same areafor a predefined amount of time (e.g., 5 seconds) an automatic captureof contextual information could occur.

In one embodiment, if the 3D video camera 102 determines that a window,open door, or the like, is in the scene, an automatic capture ofcontextual information could be performed. The automatic capture couldbe a higher resolution photo, a higher resolution video, a soundrecording or any combination thereof.

In one embodiment, if the 3D video camera 102 determines that a sound isaudible in the scene, an automatic capture of contextual informationcould be performed. In one embodiment, the sound could be one of apredefined number of sounds such as a siren, train whistle, enginenoise, voice command, etc. In one embodiment, the automatic capture ofcontextual information (e.g., a sound recording) could be performed whenany sound is discerned. Although a number of different automatic capturemethods are discussed, it should be appreciated that there are otherways that could be predefined as reasons for automatic capture ofcontextual information. Moreover, any or all of the automatic contextualinformation capture features could be manually, remotely or otherwiseturned on or off.

At 225 of FIG. 2, the captured contextual information is tagged. Ingeneral, once the user tags something in the scene during capture anumber of actions are possible. The tagging action can include, but arenot limited to, a picture taken using a camera directly available to themobile device 110 (such as a built-in or attached RGB camera) and itspose at the time of capture spatially embedded in the 3D scene; a 3Dpoint in the scene is identified and tagged as important (e.g., a pointof importance), a 3D point in the scene is identified and tagged using avoice-note that the user can record on the mobile device 110; a plane,cylinder, cone or other geometric primitive or parametric surface isfitted and linked to the 3D data under the crosshair; a geometric linkfor a plurality of points to establish a drift correction for the 3Dimagery; a measurement of a distance from a first point in the userinteractive version of the 3D scene to a second point in the userinteractive version of the 3D scene; a measurement from one scenefeature to another is taken and linked to the measurement end-points;and the like.

In one embodiment, the contextual information capturing device is in aknown orientation and position with respect to the 3D video camera 102(e.g., a known spatial relationship). For example, the contextualinformation capturing device could be fixedly mounted to the 3D videocamera 102, coupled in a housing that also contains the 3D video camera102, offset a predetermined distance and relative orientation from the3D video camera 102, or the like. In one embodiment, the contextualinformation capturing device is separate from the 3D video camera 102and when the contextual information is obtained, the 3D video camera 102would note its pose information while the contextual informationcapturing device would determine its own pose information, orientationinformation. In so doing, the data associate with the added contextualinformation would include the pose information, such that when thecontextual information is visualized in the 3D scene it would beproperly located and oriented. This orientation and position informationwould be a capturing device pose.

Once the contextual information is tagged, the flow returns to 210 wherethe determination as to whether there is any interesting contextualinformation that has not been tagged is repeated. If additionalcontextual information is recognized as not yet being tagged, the loopof 220 and 225 repeat. Once all contextual information is tagged, theloop ends and the flow continues to 230.

At 230 of FIG. 2, the capture of information is finished. In general,the reason for utilizing a secondary source to provide contextualinformation is that utilizing lower resolution on the 3D imagery reducesthe amount of data that needs to be processed in order to build the 3Dmodel. That is, the amount of data captured by the 3D video camera 102can be significantly reduced while the gist of the 3D scene is captured.In so doing, the 3D imagery can be captured and the 3D model can bebuilt using the computational capabilities of the mobile device innear-real time, there is no need for outside processing, for sendingreams of data to a more powerful machine for processing, for waiting forthe development of the 3D model, etc.

At 235 of FIG. 2, the information is saved and embedded into the 3Dmodel. A global 3D point in the scene corresponding to the touch/clickpoint can be obtained by q=P*r(c)*d where q is the 3D scene point, P isthe camera pose matrix, d is the direction vector, and r is the depthmap that yields a scalar depth given an input pixel coordinate. Forexample, the pixel position of a global 3D point in an embedded 2Dpicture can be determined by i=cam(proj(P−1*q)) where q is the global 3Dscene point, P is the camera pose matrix of a 2D embedded picture, i isthe position of the global 3D scene point in pixel coordinates in theembedded 2D picture, cam( ) is a function that takes normalized cameracoordinates to pixel coordinates and proj( ) is a function that dividesa vector by its last component.

Using the point depth a=(P−1*q)z and the pixel coordinate i theocclusion is determined by comparing a to r(i). If a<r(i) then the pointis visible from the embedded 2D picture, otherwise the point is occludedby the embedded picture content.

With reference now to FIG. 3, a flowchart 300 for displaying captured 3Dscenes augmented with contextual information is shown in accordance withan embodiment. For example, as the 3D scene is rendered, any contextualinformation that has been added to the scene can be shown as icons,links, etc. embedded in the 3D scene such as shown in FIGS. 4E through4K.

In one embodiment, the 3D scene is rendered as usual (e.g. as a pointcloud or mesh or using image-based rendering techniques) with a 3Dgraphics library (such as OpenGL or Direct3D) making sure a valid Zbuffer is retained after the last scene drawing call finishes (butbefore the rendering is presented to screen—e.g. by a swap-bufferscommand). Then the contextual scene information (or abstractionsthereof, e.g. symbols) is rendered into the same buffer with the same 3Dgraphics library, using the Z buffer for determining occlusion.

In another embodiment, the contextual scene information may be renderedfirst or in between the commands that drive the scene rendering as longas a valid Z buffer is maintained at all times. In yet anotherembodiment, the scene may be rendered in a first drawing pass withoutthe contextual information and in a second drawing pass the contextualinformation may be rendered “on top”, provided that the first drawingpass left a valid Z buffer for determining occlusion information or thatthe graphics library has the capability to run occlusion queries.

In so doing, the composition of the first and second drawing pass into afinal rendering may happen within the graphics library within a UIframework that also uses the graphics library (such as Qt Quick). Inthis case the first drawing pass would draw into an off-screen surfacethat would then be used in the final composition. Or it may happenoutside the graphics library by a compositor provided by the operatingsystem.

With reference to FIG. 4E, a screen shot 420 of a 3D scene preview priorto optimization is presented on the display 116 in accordance with anembodiment. In general, screen shot 420 includes a model preview that isnot optimized and a number of optional action icons in a ribbon. Theactions include optimize 421, measure 471, edit 423, annotate 481,screen shot 425, share 426, file 427, reference frame 461, view 431 andsettings 429.

In general, edit 423 allows lower quality points to be edited out, thecropping out of areas that are not of interest, and the like; screenshot 425 captures data, annotations, measurements, etc.; share 426allows the data to be shared as screen shots, dp files, etc. and viaemail, drop box or the like; file 427 is where the data is saved, andsettings 429 are different user adjustable settings. The actions:optimize 421, measure 471, annotate 481, reference frame 461, and view431 are described and shown in additional detail in the remaining FIGS.4F-4K and the discussion of FIG. 3. Further, although a specified numberand type of actions are shown, it should be appreciated that there couldbe more, fewer, or different actions displayed across one or more ofFIGS. 4E-4K.

At 305 of FIG. 3, the augmented and rendered 3D model is displayed. Thatis, the optimize 421 action has been selected and the 3D model has beenoptimized. In general, optimizing the scene will address driftcorrection and utilize features such as keyframes to improve theaccuracy, color, and general quality of the 3D data.

At 310 of FIG. 3, the user alters the 3D scene structure such as byusing a global scene optimization feature such as optimize 421. At 315,the positions and poses of the contextual information are updatedaccording to the change in the 3D scene structure.

Optimization

In general, during 3D video capture, as the 3D video camera 102 movesabout a room, down a hall, up some stairs, and the like, there is anamount of error or drift that can be introduced. However, usingkeyframes, geometric features, and the like, the amount of error thatoccurs in the 3D model can be reduced or even removed (e.g., optimized).

For example, by using keyframes as described in U.S. Pat. No.9,332,243B2 issued on May 3, 2016, which is hereby incorporated byreference in its entirety, there are captured 3D images that areconsidered keyframes. For example, the first image captured would bekeyframe Alpha. If the capture rate is 30 frames per second (fps),depending upon the speed of camera movement, there may be a number offrames that contain information very similar to that of keyframe Alpha.However, at some point in the movement of the 3D video camera 102 therewill be a percentage of information that is new, e.g., not in keyframeAlpha. In general, the percentage is user adjustable or fixed, but forpurposes of the discussion an exemplary 37% will be utilized.

When the new information in the frame reaches or exceeds 37% that framewill be identified as a keyframe, e.g., keyframe Bravo. This process cancontinue throughout the 3D video camera 102 operation such that anynumbers of keyframes are obtained. For example, a 3D scan could include5,000 total frames and 500 keyframes. To reduce the file size (e.g.,memory requirements), processing requirements, and the like, only the500 keyframes are kept and the remaining 4,500 frames are discarded. Inone embodiment, the 4,500 non-keyframes could be discarded after thescene has been completely scanned.

In another embodiment, the non-keyframes are discarded as soon as a nextkeyframe is reached. For example, if there are 100 frames betweenkeyframe Alpha and the identification of keyframe Bravo, as soon askeyframe Bravo is identified, the 100 in-between frames are discarded.In so doing, the memory requirements are significantly reduced.Moreover, the amount of measurement error is also reduced as themeasurement error is now between keyframes instead of each and everycaptured 3D image. In so doing, the rendering of the 3Dscene/environment/model will be performed using only the keyframes. Inanother embodiment, the non-keyframes are discarded as soon as theirnon-keyframe-ness is determined by the program and their data has beenused by the program e.g. to alter, refine or extend other existingkeyframes.

Similarly, geometric features can be used to correct errors in the 3Dimagery. For example, if the 3D video camera 102 is going up stairs, thelanding of floor 1 could be tagged as being parallel to the landing offloor 2, the landing of floor 3, etc. In so doing, the 3D model would bepulled into alignment based on the indications that the landings at eachfloor are parallel and hence drift would be minimized. As such, insteadof having a 14 story drift that causes a significant lean to the 3Dmodel, there would only be minimal floor-to-floor drift which would beautomatically corrected at each landing. This tagging could similarly beused on parallel walls, wall-to-floor intersections, etc.

In one embodiment, to transform the contextual information (e.g., 2Dimagery) into the 3D scene, a link is used to tie the 2D imagery to thebest keyframe. E.g., the keyframe from which the point is visible basedon closeness of the point to the optical center of the keyframe and thesurface angle under which the point appears in the keyframe and thepresence of nearby points in the keyframe point cloud is obtained. The3D point is then expressed in keyframe camera coordinates and convertedback to global coordinates after the structural change has happened(which may of course have altered the keyframe pose). Changes in thekeyframe point cloud are also taken into account and the depth of thetagged point is adjusted accordingly in camera coordinates beforeconverting back to global coordinates.

With reference again to 305 and now to FIG. 4F, a screen shot 430 of anoptimized 3D scene including a view 431 tool bar is presented on display116 in accordance with an embodiment. In general, screen shot 430illustrates the optimized 3D model and a number of optional action iconsprovided in a tool bar after the selection of view 431. The actionsinclude surface 432, points 433, camera frusta 434, frames 435, home436, and up arrow 437. In general, home 436 and up arrow 437 areoptional/adjustable/re-definable actions. For example, in oneembodiment, up arrow 437 would return to the previous screen while home436 would return to the main menu, a fresh capture set-up, or the like.

In one embodiment, surface 432 provides the display of the 3D model assurfaces such as the wall and cabinet. In contrast, points 433 willdisplay the 3D model as a collection of individual 3D points with eachpoint sized dependent on the display screen resolution and a userselected value. Camera frusta 434 will show the location of thecontextual information (including the pose) in the 3D model as shown anddescribed in further detail in the discussion of FIG. 4G. Frames 435provides a side bar to the 3D model that includes a preview of thecontextual information which is shown and described in further detail inthe discussion of FIG. 4H.

At 320 of FIG. 3, the user touches/clicks/or otherwise selects acontextual information identifier in the rendered 3D scene. For example,referring to FIG. 4G, a screen shot 440 of an optimized 3D scenerendered with contextual information identifiers 441-443 presented onthe display 116 is shown in accordance with an embodiment. For purposesof clarity, only the aspects of screen shot 440 of FIG. 4G that aredifferent than those of FIG. 4E are described. In screen shot 440, thecontextual information identifiers 441-443 are displayed within the 3Dmodel environment.

Contextual information identifiers 441-443 images could include alocation, an orientation of the captured contextual information, etc. Inone embodiment, if a viewer of the 3D scene wants to see what isincluded in the contextual information, they would select one of thecontextual information identifiers 441-443 and then the contextualinformation that was captured would be displayed.

For example, the 3D scene could be an electrical room and the contextualinformation is a high resolution photo of a fuse box that would providedetail and resolution about the contents of the fuse box, labels on thefuse box, information written in the fuse box, etc. In anotherembodiment, the 3D scene could be a crime scene and the contextualinformation could be a sound recording from an answering machine that islinked to the answering machine in the 3D scene. In yet anotherembodiment, the 3D scene could be an engine room and the contextualinformation could include a sound recording and a high resolution videosnippet that would provide operating sounds to allow a mechanic toevaluate the operation of an engine therein.

The 3D scene could include any number of contextual informationidentifiers 441-443. The user could select one to access the underlyingcontextual information, close the contextual information to return tothe 3D scene and then select another of the contextual informationidentifiers 441-443. Although three contextual information identifiers441-443 are shown in FIGS. 4G and 4H, the use of three contextualinformation identifiers is provided merely for purposes of clarity.

At 330 of FIG. 3, the virtual scene camera “flies” to the viewpoint fromwhich the contextual information was captured. In one embodiment, asshown in screen shot 440 of FIG. 4G, the contextual information isdisplayed within the 3D model. For example, when the user selects one ofcontextual information identifiers 441-443, the virtual camera used forthe rendering of the 3D scene would “fly” from its current position intothe pose of the contextual information within the 3D model such the endof the camera flight that the virtual 3D capture device has assumed thepose associated with the selected contextual information.

In one embodiment, a special 2D/3D visualization technique is providedfor the case when the user selects an embedded 2D picture and thevirtual scene camera “flies” from its current viewpoint to the viewpointthat the picture was taken from. In that case, the rendering of the 3Dmodel gradually blends into a rendering of the 2D picture when thevirtual scene camera pose is in the vicinity of the embedded picturepose, taking into account parallax effects due to varying depth, suchthat when the virtual camera reaches its destination position theaugmented rendering of the 3D model becomes equal to the 2D picture.

Thus, the selection of the fuse box contextual information identifierwould present the higher resolution image of the fuse box within the 3Dscene to maintain the context, spatial information, and the like for theuser viewing the 3D scene.

By embedding the contextual information (e.g., a 2D higher resolutionimage) into the 3D model, it also allows 3D information, (e.g., depth,angle, etc.) to be obtained from the 2D higher resolution image. Forexample, since the 3D imagery is underlying the 2D image, thedimensional data from the underlying 3D imagery can be mapped to thepixels in the overlaid 2D image.

For example, by using 3D model rendering and based on the embedding ofthe 2D image into the 3D model, any dimensional data from the 3D modelthat falls within a scope of the 2D image is tied to the 2D imagethereby generating a depth-mapped 2D image. The depth-mapped 2D imageallows the dimensional data to be available when the depth-mapped 2Dimage is presented outside of the 3D model (e.g., depth measurements canbe obtained from the 2D image). Such as, if the 2D image is selectedfrom the list and displayed alone, shared, emailed, or the like.

At 325 of FIG. 3, the user selects the contextual information from alist such as frames 451-453. For example, with reference now to FIG. 4H,a screen shot 450 of an optimized 3D scene rendered with contextualinformation identifiers 441-443 and their associated frames 451-453 arepresented on the display 116 in accordance with an embodiment. Forpurposes of clarity, only the aspects of screen shot 450 of FIG. 4H thatare different than those of FIG. 4G are described. In general, screenshot 450 is obtained when frames 435 of FIG. 4F is selected. In oneembodiment, when frames 435 is selected, any contextual information thatwould be found in the 3D scene are provided in a side bar to the 3Dmodel that includes a preview of the contextual information. Forexample, frame 452 is the image that is indicated by contextualinformation identifier 442, frame 451 is the image that is indicated bycontextual information identifier 441, and frame 453 is the image thatis indicated by contextual information identifier 443. Although oneversion of the frames 451-453 layout is shown it should be appreciatedthat the visual layout of frames 451-453 may take several forms (e.g.vertical/horizontal lists, a 2D grid of items, etc.).

In one embodiment, one piece of contextual information could bedisplayed as stand-alone information. For example, when the user selectsa contextual information identifier or frame (e.g.,double-clicks/double-taps a contextual information identifier or frame,etc.) the contextual information would be presented to the user asstand-alone data outside of the 3D model. Thus, such selection of thefuse box contextual information would open the higher resolution imageon the full display of the mobile device and outside of the 3D model.However, even though the 2D image would be displayed outside of the 3Dmodel, since the 2D image has been mapped to the 3D model, themeasurement information would still be available.

Referring now to FIG. 4I, a screen shot 460 of an optimized 3D scenewith an added reference coordinate system 465 presented on the display116 is shown in accordance with an embodiment. In one embodiment, screenshot 460 is reached by selecting the reference frame 461 icon from thetop ribbon. Although a number of screen menus and operation/activitynames are provided herein, the naming and menus are exemplary. That is,the location of the operation/activity within the menu system and thename of the operation/activity could be amended during softwarerevisions, based on user feedback, or for many other legal, utilitarianor other reasoning. For example, “reference frame” could also be called“coordinate system” and found under the “Edit” menu items, etc.

After reference frame 461 icon is selected, the use is able to referencethe origin point or any known point in the scene, e.g., point 464. Inone embodiment, the user sets the origin (of the reference system) byselecting a “set origin” option and then holding a point within the 3Dscene, e.g., holding a floor point 463. After floor point 463 isselected, the user can select the primary z-axis while holding floorpoint 463 which will set the z-axis to be directed straight up from thefloor. The user can then tap to set the x-axis as secondary axis andthen tap-and-hold a point on the wall 462 to set the x-axis straight outfrom the wall. This will establish a suitably oriented xyz referencesystem at point 464. Once the reference system 465 is established, all3D data can automatically be referenced to that selected coordinatesystem.

Referring now to FIG. 4J, a screen shot 470 of an optimized 3D sceneshowing an in-scene measurement capability 474 is presented on thedisplay 116 in accordance with an embodiment. In one embodiment, screenshot 470 is reached by selecting the measurement 471 icon from the topribbon. After measurement 471 icon is selected, the use is able to takemeasurements from within the 3D scene.

In one embodiment, to take a measurement the user taps a first location472 in the 3D scene. After first location 472 is selected, the user cantap (or select) a second location 473. Once both locations are selected,the measurement will be automatically determined. In general, themeasurement capability 474 includes a total distance between the pointsas well as the change in the x, y, and z components in the currentlyactive reference system. In one embodiment, the first location 472 andsecond location 473 are adjustable such as via dragging visual pointselectors or the like.

At 333 of FIG. 3, the user determines whether they need any additionaldata regarding the contextual information in the rendered 3D scene. Ifthey do not need any additional information, then the flowchart 300 iscomplete. If the user does need additional data, at 340 the user willre-select the contextual information.

Referring now to 350 it is determined if the contextual information isan embedded 2D image. If it is not a 2D image, then at 360 thecontextual information (e.g., higher resolution video, sound recording,etc.) is presented on the screen or played as a sound and the flow chartloops back to 305.

However, if the contextual information is an embedded 2D image, thenreferring now to 353 of FIG. 3, the image is displayed in a zoomable anddraggable view that allows annotation. At 355 the user decides if theywant to annotate. If they do not want to annotate the image then theyare finished.

If the user does want to annotate the 2D image, then at 358 the userannotates the image and the annotation point is added in globalcoordinates and becomes a contextual data item and the action isfinished. For example, with reference now to FIG. 4K, a screen shot 480of an optimized 3D scene with added annotation menu 484 presented on thedisplay 116 is shown in accordance with an embodiment. In oneembodiment, screen shot 480 is reached by selecting annotate 481 iconfrom the top ribbon. After annotate 481 icon is selected, the use isable to provide notes or the like into the 3D scene, such as annotation482 and annotation 483. Although screen shot 480 shows the annotation ofa 3D scene within a visualization of the 3D scene, it should beappreciated that the annotation mechanisms work regardless of whetherthe 3D scene is displayed using a virtual viewpoint (as in screen shot480) from which the scene is rendered or whether the scene is displayedas a depth-mapped 2D image (as discussed in [0095]).

In one embodiment, to make an annotation in the 3D scene the user tapsthe add function in annotation menu 484. After add is selected, the usercan hold a point within the 3D scene to bring up a magnifying window,and then drag to target a point in the scene once the user releases thepoint the annotation (e.g., annotation 482 and/or 483) can be placed. Inone embodiment, the annotation can be written and is added via anonscreen keyboard, an actual keyboard, voice-to-text input, or the like.In one embodiment, the annotation can include a caption (e.g., title)and then additional detail information. For example, annotation 482could be captioned “White Wall” and the additional detail informationcould include “painted, drywall, etc.”. In one embodiment, only thecaption of annotation 482 is shown in the 3D model and the additionaldetail information is provided when the annotation 482 is selected bythe user.

In one embodiment, annotation can be added to the 3D model, tocontextual information that is within the 3D model, etc. For example, ifthe 2D picture is a picture of a puddle from a leak, the 2D picturecould be annotated. That annotation would be found within the 2D imagedata. When the 2D image is embedded within the 3D model, the annotatedinformation from the 2D image would present as annotated information inthe 3D model. That is, through the spatial embedding of the 2D picturesthe 3D coordinates of the picked points (e.g., tags, annotations,measurements, etc.) can be inferred automatically.

In one embodiment, tags, annotations, measurements or otheraugmentations of the 3D scene are not only displayed as part of the 3Dmodel rendering but also as part of the 2D picture display, enabled bythe spatial embedding of the 2D pictures in the scene. In so doing, 3Dpoints selected or annotated through one 2D picture not only becomevisible in the 3D model rendering but also become visible in all other2D pictures that have direct visibility of the 3D point.

Thus, by embedding contextual information such as high-resolution 2Dphotos in a 3D model and giving the viewer of the 3D model the abilityto seamlessly “fly” between different embedded 2D pictures in the scenesolves the problem of regular 2D pictures lacking 3D visual context. Inother words, the user is able to get and maintain a spatialunderstanding of where the pictures were taken and how they relate toeach other in 3D.

Further, by presenting a zoomable and draggable 2D picture displayinterface for displaying the embedded 2D pictures, the user can workwith a picture interface that's already familiar from other applicationswithout losing the 3D context of the photo. Similarly, placing tags,annotations or measurements in the 3D model or performing any otheroperation that requires the user to pick points with a physical 3Dmeaning can be done through a familiar 2D picture interface. This solvesthe problem of plain 2D image collections not having depth informationor any other kind of spatial relation to the captured scene. Moreoverthe 2D interface provides increased visual detail, a more familiar userinterface, increased point picking precision as compared to selectingthe points in a rendering of the 3D model in case a 3D model isavailable.

Moreover, giving a collection of contextual information a spatialembedding in a 3D model also solves the problem of annotation/tagvisibility across different contextual information, without the userhaving to re-identify the same points across any number of differentcontextual information within the collection. The problem ofcommunicating the 3D positions and orientations of additional rich sceneinformation is also solved by augmenting and/or overlaying the renderingof a 3D model with abstractions (e.g. icons, symbols) of the embeddedadditional scene information.

In addition, letting the user tag/augment the 3D scene during captureusing a visualization of the live capture process+crosshair provides auser-friendly alternative to tagging/augmenting the 3D scenepost-capture (using an interface based on a rendering of the captured 3Dmodel). Since the user is already present in the scene during captureand is controlling a capture device, singularizing a particular featurein the scene becomes as easy as shooting pictures with a digital camera.This is in contrast to augmenting the scene by operating with a full(often bulky) 3D model after capturing.

With reference now to FIG. 5, portions of the technology for providing acommunication composed of computer-readable and computer-executableinstructions that reside, for example, in non-transitorycomputer-readable storage media of a computer system. That is, FIG. 5illustrates one example of a type of computer that can be used toimplement embodiments of the present technology. FIG. 5 represents asystem or components that may be used in conjunction with aspects of thepresent technology. In one embodiment, some or all of the componentsdescribed herein may be combined with some or all of the components ofFIG. 5 to practice the present technology.

FIG. 5 illustrates an example computer system 500 used in accordancewith embodiments of the present technology such as mobile device 110. Asshown in FIG. 5, computer system 500 of FIG. 5 is well adapted to havingperipheral computer readable media 502 such as, for example, a disk, acompact disc, a flash drive, and the like coupled thereto.

Computer system 500 of FIG. 5 includes an address/data/control bus 504for communicating information, and a processor 506A coupled to bus 504for processing information and instructions. As depicted in FIG. 5,system 500 is also well suited to a multi-processor environment in whicha plurality of processors 506A, 506B, and 506C are present. Conversely,system 500 is also well suited to having a single processor such as, forexample, processor 506A. Processors 506A, 506B, and 506C may be any ofvarious types of microprocessors. Computer system 500 also includes datastorage features such as a computer usable volatile memory 508, e.g.,random access memory (RAM), coupled to bus 504 for storing informationand instructions for processors 506A, 506B, and 506C.

System 500 also includes computer usable non-volatile memory 510, e.g.,read only memory (ROM), coupled to bus 504 for storing staticinformation and instructions for processors 506A, 506B, and 506C. Alsopresent in system 500 is a data storage unit 512 (e.g., a magnetic diskdrive, optical disk drive, solid state drive (SSD), and the like)coupled to bus 504 for storing information and instructions. Computersystem 500 also includes an optional alpha-numeric input device 514including alphanumeric and function keys coupled to bus 504 forcommunicating information and command selections to processor 506A orprocessors 506A, 506B, and 506C. Computer system 500 also includes anoptional cursor control device 516 coupled to bus 504 for communicatinguser input information and command selections to processor 506A orprocessors 506A, 506B, and 506C. Optional cursor control device may be atouch sensor, gesture recognition device, and the like. Computer system500 of the present embodiment also includes an optional display device518 coupled to bus 504 for displaying information.

Referring still to FIG. 5, optional display device 518 of FIG. 5 may bea liquid crystal device, cathode ray tube, OLED, plasma display deviceor other display device suitable for creating graphic images andalpha-numeric characters recognizable to a user. Optional cursor controldevice 516 allows the computer user to dynamically signal the movementof a visible symbol (cursor) on a display screen of display device 518.Many implementations of cursor control device 516 are known in the artincluding a trackball, mouse, touch pad, joystick, non-contact input,gesture recognition, voice commands, bio recognition, and the like. Inaddition, special keys on alpha-numeric input device 514 capable ofsignaling movement of a given direction or manner of displacement.Alternatively, it will be appreciated that a cursor can be directedand/or activated via input from alpha-numeric input device 514 usingspecial keys and key sequence commands.

System 500 is also well suited to having a cursor directed by othermeans such as, for example, voice commands. Computer system 500 alsoincludes an I/O device 520 for coupling system 500 with externalentities. For example, in one embodiment, I/O device 520 is a modem forenabling wired or wireless communications between system 500 and anexternal network such as, but not limited to, the Internet or intranet.A more detailed discussion of the present technology is found below.

Referring still to FIG. 5, various other components are depicted forsystem 500. Specifically, when present, an operating system 522,applications 524, modules 526, and data 528 are shown as typicallyresiding in one or some combination of computer usable volatile memory508, e.g. random access memory (RAM), and data storage unit 512.However, it is appreciated that in some embodiments, operating system522 may be stored in other locations such as on a network or on a flashdrive; and that further, operating system 522 may be accessed from aremote location via, for example, a coupling to the internet. In oneembodiment, the present technology, for example, is stored as anapplication 524 or module 526 in memory locations within RAM 508 andmemory areas within data storage unit 512. The present technology may beapplied to one or more elements of described system 500.

System 500 also includes one or more signal generating and receivingdevice(s) 530 coupled with bus 504 for enabling system 500 to interfacewith other electronic devices and computer systems. Signal generatingand receiving device(s) 530 of the present embodiment may include wiredserial adaptors, modems, and network adaptors, wireless modems, andwireless network adaptors, and other such communication technology. Thesignal generating and receiving device(s) 530 may work in conjunctionwith one or more communication interface(s) 532 for coupling informationto and/or from system 500. Communication interface 532 may include aserial port, parallel port, Universal Serial Bus (USB), Ethernet port,Bluetooth, thunderbolt, near field communications port, WiFi, Cellularmodem, or other input/output interface. Communication interface 532 mayphysically, electrically, optically, or wirelessly (e.g., via radiofrequency) couple computer system 500 with another device, such as amobile phone, radio, or computer system.

System 500 also includes one or more 2D and/or 3D sensor input(s) 545coupled with bus 504 for enabling system 500 to capture 2D and/or 3Dinformation from sensors attached to system 500 and/or other electronicdevices and computer systems. 2D and/or 3D sensor input(s) 545 of thepresent embodiment may include 2D and/or 3D audio, video, and/or stillcapture technology such as (or similar to) that described in FIGS. 1-4Kherein or their equivalents.

The computing system 500 is only one example of a suitable computingenvironment and is not intended to suggest any limitation as to thescope of use or functionality of the present technology. Neither shouldthe computing environment be interpreted as having any dependency orrequirement relating to any one or combination of components illustratedin the example computing system 500.

The present technology may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, etc., that performparticular tasks or implement particular abstract data types. Thepresent technology may also be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer-storage media including memory-storage devices.

The foregoing Description of Embodiments is not intended to beexhaustive or to limit the embodiments to the precise form described.Instead, example embodiments in this Description of Embodiments havebeen presented in order to enable persons of skill in the art to makeand use embodiments of the described subject matter. Moreover, variousembodiments have been described in various combinations. However, anytwo or more embodiments may be combined. Although some embodiments havebeen described in a language specific to structural features and/ormethodological acts, it is to be understood that the subject matterdefined in the appended claims is not necessarily limited to thespecific features or acts described above. Rather, the specific featuresand acts described above are disclosed by way of illustration and asexample forms of implementing the claims and their equivalents.

What is claimed is:
 1. An assembly comprising: a 3D capture device tocapture 3D imagery at a first resolution; and a mobile computing devicecoupled to the 3D capture device, the mobile computing devicecomprising: a memory; a contextual information capturing device, thecontextual information capturing device to capture at least one piece ofcontextual information, the at least one piece of contextual informationincluding a capture location data and a pose data of the at least onepiece of contextual information at a time of capture; a processor, theprocessor to: receive the 3D imagery from the 3D capture device; rendera 3D model using the 3D imagery; utilize the capture location data andthe pose data to embed the at least one piece of contextual informationinto an appropriate location in the 3D model; and an interactivedisplay, the interactive display to present a user interactive versionof the 3D model including the embedded at least one piece of contextualinformation.
 2. The assembly of claim 1 where the 3D model is renderedin real-time or near real-time.
 3. The assembly of claim 1 where theprocessor is further to: determine a plurality of keyframes from the 3Dimagery; discard any frames in the 3D imagery that are not keyframes;and render the 3D model with only the plurality of keyframes to reduce afile size and a processing requirement for the render of the 3D model.4. The assembly of claim 3 where the processor is further to: tie the atleast one piece of contextual information to a best keyframe of theplurality of keyframes, the best keyframe being selected from theplurality of keyframes using a metric comprising: a distance of akeyframe optical center position to a pose position of the one piece ofcontextual information; an angular difference between a keyframeorientation and a pose orientation of the one piece of contextualinformation; and a 3D coordinates of a capture location data of the onepiece of contextual information as seen from a viewpoint of thekeyframe.
 5. The assembly of claim 1 wherein the contextual informationcapturing device is a 2D camera, and the at least one piece ofcontextual information is a 2D image having a second resolution equal toor higher than the first resolution.
 6. The assembly of claim 5 wherethe processor is further to: map, based on a spatial embed of the 2Dimage into the 3D model, any dimensional data from the 3D model thatfalls within a scope of the 2D image into the 2D image to generate apoint-mapped 2D image, the point mapping allowing the dimensional datato be available when the 2D image is presented independently of the 3Dmodel.
 7. The assembly of claim 1 wherein the contextual informationcapturing device is a 2D video camera, and the at least one piece ofcontextual information is a 2D video snippet having a second resolution.8. The assembly of claim 1 wherein the contextual information capturingdevice is a sound recorder and the at least one piece of contextualinformation is an audio snippet represented as a digitized waveform. 9.A method, comprising: capturing, with a 3D capture device, a pluralityof 3D images at a first resolution; capturing, with a component on amobile computing device, at least one piece of contextual information,the at least one piece of contextual information including a capturelocation data and a pose data of the at least one piece of contextualinformation at a time of capture; receiving, and storing in a memory ofthe mobile computing device, the plurality of 3D images from the 3Dcapture device; rendering, via a processor of the mobile computingdevice, the plurality of 3D images into a 3D model; embedding, via theprocessor and using the capture location data and the pose data, the atleast one piece of contextual information into a correct location in the3D model; and presenting, via an interactive display of the mobilecomputing device, a user interactive version of the 3D model, the userinteractive version of the 3D model including the embedded at least onepiece of contextual information.
 10. The method of claim 9, wherein therendering further comprises: determining a plurality of keyframes fromthe plurality of 3D images; discarding any of the plurality of 3D imagesthat are not keyframes; and rendering the 3D model with only theplurality of keyframes, to reduce a file size, a processing requirement,and a memory requirement for the rendering of the 3D model.
 11. Themethod of claim 10 further comprising: linking the at least one piece ofcontextual information to a best keyframe of the plurality of keyframes,the best keyframe being selected from the plurality of keyframes using ametric comprising: a distance of a keyframe optical center position to apose position of the one piece of contextual information; an angulardifference between a keyframe orientation and a pose orientation of theone piece of contextual information; and a 3D coordinates of a capturelocation data of the one piece of contextual information as seen from aviewpoint of the keyframe.
 12. The method of claim 10 wherein theembedding of the at least one piece of contextual information into acorrect location in the 3D model comprises: displaying an icon spatiallyembedded in the 3D model representing the at least one piece ofcontextual information, the icon in the correct location; and utilizingthe pose data for the at least one piece of contextual information toorient the icon in the 3D model into a correct as taken orientation. 13.The method of claim 10 wherein the component is selected from the groupconsisting of: a 2D camera, and the at least one piece of contextualinformation is a 2D image having a second resolution equal to or higherthan the first resolution, a 2D video camera and the at least one pieceof contextual information is a 2D video snippet having a thirdresolution equal to or higher than the first resolution, and a soundrecorder and the at least one piece of contextual information is anaudio snippet represented as a digitized waveform.
 14. The method ofclaim 13 further comprising: mapping, based on a spatial embedding ofthe 2D image into the 3D model, any dimensional data from the 3D modelthat falls within a scope of the 2D image into the 2D image to generatea point-mapped 2D image, the point-mapped 2D image allowing thedimensional data to be available when the 2D image is presentedindependently of the 3D model.
 15. The method of claim 10 furthercomprising: adding an annotation to the user interactive version of the3D model.
 16. The method of claim 10 further comprising: tagging atleast one point in the user interactive version of the 3D model, whereinthe tagging is used for an action from the group consisting of:identifying a point of importance, adding a voice-note to a point,geometrically linking a plurality of points to establish a geometricconstraint that can be used in a global 3D model optimization processwherein a 3D model accuracy and appearance is improved according to aconfigurable criteria, and measuring a distance from a first point inthe user interactive version of the 3D model to a second point in theuser interactive version of the 3D model.
 17. The method of claim 10further comprising: automatically capturing, with the component on themobile computing device, the at least one piece of contextualinformation.
 18. A mobile computing device comprising: a memory; a 3Dcapture device to capture 3D imagery at a first resolution; a contextualinformation capturing device to capture a plurality of pieces ofcontextual information, each of the plurality of pieces of contextualinformation including a capture location data and a pose data of the atleast one piece of contextual information at a time of capture; one ormore processors, the one or more processors to: determine a plurality ofkeyframes from the 3D imagery; discard any frames in the 3D imagery thatare not keyframes; render a 3D scene with only the plurality ofkeyframes to reduce a file size, a processing requirement, and a memoryrequirement for the render of the 3D scene; utilize the capture locationdata and the pose data to embed each of the plurality of pieces ofcontextual information into a correct location in the 3D scene; and aninteractive display to present a user interactive version of the 3Dscene, the user interactive version of the 3D scene including theembedded plurality of pieces of contextual information.
 19. The mobilecomputing device of claim 18, wherein the contextual informationcapturing device is a 2D camera, the plurality of pieces of contextualinformation are 2D images having a second resolution equal to or higherthan the first resolution, and where the one or more processors arefurther to: map, based on a spatial embed of the 2D images into the 3Dscene, any dimensional data from the 3D scene that falls within a scopeof each of the 2D images into each of the 2D images to generate aplurality of point-mapped 2D images, the plurality of point-mapped 2Dimages allowing the dimensional data to be available when one or more ofthe plurality of point-mapped 2D images are presented independently ofthe 3D scene.
 20. The mobile computing device of claim 18, where the oneor more processors are further to: tag at least one point in the userinteractive version of the 3D scene, wherein the tag is used to performan action from the group consisting of: an identification of a point ofimportance in the 3D scene, an addition of a voice-note to the at leastone point, a geometric link for a plurality of points to establish ageometric constraint that can be used in a global 3D scene optimizationprocess wherein a 3D scene accuracy and appearance is improved accordingto a configurable criteria, and a measurement of a distance from a firstpoint in the user interactive version of the 3D scene to a second pointin the user interactive version of the 3D scene.